Pull Request Summary

Major refactor of W3SRCE subroutine to calculate source terms for arrays of spectra rather than a single spectrum.
Intended to facilitate acceleration on GPU architectures. See presentation in discussion #736 for more details.

Description

The purpose of this change is to address the poor performance of the WW3 source term calculations when running on a GPU. The current implementation of the source term routines processes a single spectrum at a time, which unfortunately does not expose sufficiently large arrays/loops to fully subscribe the large number of parallel threads available on a GPU resulting in poor GPU performance.

The proposed solution to this is to refactor the source term packages to process arrays of spectra, rather than a single spectrum. This provides a large independent seapoint dimension to fully utilise the available GPU threads.

The first step in achieving this is to refactor the W3SRCE routine to accept arrays with a seapoint dimension, rather than values for a single seapoint. This moves the NSEA loop one level down from W3WAVE into W3SRCE. This will allow for the second stage of this refactoring (which will happen at a later date) which will allow individual source term routines to be parallelised over multiple seapoints. However, for this change, source term subroutine calls will be wrapped in a seapoint loop to maintain their current functionality.

This first set of changes does not add any GPU acceleration - it is merely paving the way to facilitate GPU acceleration in the next phase. There should be no change to the model outputs and the performance impact on the CPU should be minimal.

There are a few points of consideration to this change:

1. It does add some extra complexity to the code (w3srce is already quite large and complex)
2. New temporary arrays are needed to gather variables on the full grid (NSEA) to contiguous arrays on the local grid (NSEAL).
3. The change in the looping structure could affect B4B reproducibility (when compared against original code) with some compilers when using higher optimization levels. I have found the Cray compiler to be particularly prone to this.

Summary of important changes:

• W3WAVE now passes full NSEA(L) dimensioned arrays to W3SRCE`
• Loop over seapoints NSEA(L) now done in W3SRCE
• W3SRCE now loops over tiles are seapoints
  • To keep source term subroutines working as before, these are wrapped in sepoint loops
  • As source term subroutines are accelerated on GPU, it is expected they will be refactored to receive arrays of spectra.
  • Array tiling implemented to mitigate impact on CPU performance (though cache misses) and maintain current functionality of processing 1 seapoint at a time on the CPU. Can be set to larger tiles for GPU parallelisation.
  • New TILE sized arrays for mapping of local seapoints to full grid seapoints. (JSEA -> ISEA).
• Various variables no longer required, or were not being used:
  • VAOLD/VAoldDummy not used in W3SRCE; removed. VAOLD used by PDLIB, but not in w3srce.
  • DELX, DELY and DELA not used. Removed.
  • REFLEC and REFLECD scalars removed - passed in REFL[CD] arrays explicitly instead. Calculation now done in w3srce
  • D50 and PSIC scalars removed. Passing whole arrays.
  • TMP[1234] scalars removed and passed in arrays direct.
  • VSIO/VDIO/SHAVEIO now optional arguments (only for PDLIB)
  • LSLOC test on VSIO/VDIO now done in w3srce.
• A new loop added to end of main chunk loop to copy local grid variables back to full grid (e.g. USTAR)
• The following 2D variables have been split into multiple 1D arrays to avoid temporary arrays being created when tiling:
  • WHITECAP
  • TAUICE
  • TAUBBL
  • BEDFORMS

Commit Message

Refactor of W3SRCE to allow source term for arrays of seapoints to be calculated rather than a single seapoint.
To facilitate later acceleration of source terms on GPU architecture.

Check list

• Branch is up to date with the authoritative repository (NOAA-EMC) develop branch.
• Checked the checklist for a developer submitting to develop.
• If a version number update is required, checked the updating version number checklist.
• If a new feature was added, a regression test for testing the new feature is added.

Testing

• How were these changes tested? Regression tests.
• Are the changes covered by regression tests? (If not, why? Do new tests need to be added?) Yes.
• Have the matrix regression tests been run (if yes, please note HPC and compiler)? Yes; Cray XC; Intel compiler.
• Please indicate the expected changes in the regression test output, (Note the list of known non-identical tests.)
• Please provide the summary output of matrix.comp (matrix.Diff.txt, matrixCompFull.txt and matrixCompSummary.txt):